Tag: GPU memory errors

ECC memory errors: rates, causes, and consequences in datacenter and AI systems

What does the evidence show about the rate, causes, and system-level consequences of ECC-protected memory errors in modern computing systems, and how effective are the mitigations?

Error-correcting-code (ECC) memory protects datacenter and AI systems from bit flips that would otherwise corrupt results, but the evidence shows the protection is partial and the threat is changing. Field studies spanning 1979-2026 find real DRAM error rates orders of magnitude above lab estimates (more than 8% of DIMMs affected per year at Google), show that most errors are hard, repeatable faults rather than cosmic-ray soft errors, and document that ECC catches most but not all of them - single-bit-correction codes can miscorrect double-bit errors, on-die ECC hides raw error patterns from operators, and GPU/HBM studies reveal error rates varying by three orders of magnitude across otherwise identical clusters. The largest caveat: the load-bearing field studies are from a handful of large operators (Google, Meta, LANL, BSC, Alibaba), and AI-specific evidence is young, with several key GPU/HBM results still preprints.

ECC memory · DRAM errors · HBM reliability · GPU memory errors · silent data corruption · row hammer · memory scrubbing · soft errors